Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

URL: Implement custom getQueryArgs, buildQueryString #20693

Merged
merged 5 commits into from
Nov 30, 2020
Merged

Conversation

aduth
Copy link
Member

@aduth aduth commented Mar 6, 2020

This pull request seeks to refactor the @wordpress/url module to use a custom implementation of parsing and serializing query strings, removing the qs dependency in the process. The intent with these changes is to have a stronger contractual guarantee of alignment of behavior between @wordpress/url and corresponding PHP functions (build_query, parse_str). It's expected this may also result in a smaller overall bundle size, as we are not using the full extent of the feature set of qs.

Implementation Notes:

I had some early hopes to be able to use URLSearchParams for what's implemented here, but it behaves quite differently than we expect from these functions, largely in two ways:

  • Spaces as + vs. as %20. URLSearchParams uses %20 (per URL Living Standard). PHP uses + (per RFC1866 form encoding).
  • Object and array values. URLSearchParams has limited support for "multiple" values (getAll, append), but these are formatted differently than they are in PHP (i.e. neither square brackets nor indices in the argument name). There's also no built-in support for "keyed" arguments.

Testing Instructions:

Verify unit tests continue to pass:

npm run test-unit

@aduth aduth added [Status] In Progress Tracking issues with work in progress [Package] Url /packages/url labels Mar 6, 2020
@github-actions
Copy link

github-actions bot commented Mar 6, 2020

Size Change: -1.28 kB (0%)

Total Size: 1.19 MB

Filename Size Change
build/a11y/index.js 1.14 kB -1 B
build/annotations/index.js 3.8 kB +2 B (0%)
build/api-fetch/index.js 3.42 kB +1 B
build/block-directory/index.js 8.72 kB -4 B (0%)
build/block-editor/index.js 128 kB -4 B (0%)
build/block-library/index.js 148 kB -5 B (0%)
build/block-serialization-default-parser/index.js 1.88 kB +8 B (0%)
build/blocks/index.js 48.1 kB +1 B
build/components/index.js 172 kB -33 B (0%)
build/compose/index.js 9.95 kB -4 B (0%)
build/core-data/index.js 14.8 kB -1 B
build/data/index.js 8.98 kB +1 B
build/date/index.js 11.2 kB -3 B (0%)
build/deprecated/index.js 768 B -1 B
build/edit-navigation/index.js 11.1 kB +5 B (0%)
build/edit-post/index.js 306 kB +1 B
build/edit-site/index.js 24.1 kB +3 B (0%)
build/editor/index.js 43.3 kB -26 B (0%)
build/element/index.js 4.63 kB +2 B (0%)
build/format-library/index.js 6.86 kB +2 B (0%)
build/hooks/index.js 2.27 kB -1 B
build/html-entities/index.js 623 B +1 B
build/i18n/index.js 3.56 kB -1 B
build/keyboard-shortcuts/index.js 2.54 kB -1 B
build/keycodes/index.js 1.93 kB -1 B
build/media-utils/index.js 5.31 kB -4 B (0%)
build/notices/index.js 1.82 kB -2 B (0%)
build/nux/index.js 3.42 kB +1 B
build/plugins/index.js 2.56 kB +1 B
build/primitives/index.js 1.43 kB -1 B
build/priority-queue/index.js 791 B +1 B
build/redux-routine/index.js 2.84 kB +3 B (0%)
build/reusable-blocks/index.js 2.92 kB -1 B
build/shortcode/index.js 1.69 kB +1 B
build/url/index.js 2.84 kB -1.22 kB (42%) 🎉
build/viewport/index.js 1.86 kB -3 B (0%)
ℹ️ View Unchanged
Filename Size Change
build/autop/index.js 2.84 kB 0 B
build/blob/index.js 665 B 0 B
build/block-directory/style-rtl.css 943 B 0 B
build/block-directory/style.css 942 B 0 B
build/block-editor/style-rtl.css 11.2 kB 0 B
build/block-editor/style.css 11.2 kB 0 B
build/block-library/editor-rtl.css 8.95 kB 0 B
build/block-library/editor.css 8.95 kB 0 B
build/block-library/style-rtl.css 8.27 kB 0 B
build/block-library/style.css 8.27 kB 0 B
build/block-library/theme-rtl.css 789 B 0 B
build/block-library/theme.css 790 B 0 B
build/block-serialization-spec-parser/index.js 3.06 kB 0 B
build/components/style-rtl.css 15.3 kB 0 B
build/components/style.css 15.3 kB 0 B
build/data-controls/index.js 827 B 0 B
build/dom-ready/index.js 571 B 0 B
build/dom/index.js 4.95 kB 0 B
build/edit-navigation/style-rtl.css 881 B 0 B
build/edit-navigation/style.css 885 B 0 B
build/edit-post/style-rtl.css 6.42 kB 0 B
build/edit-post/style.css 6.4 kB 0 B
build/edit-site/style-rtl.css 3.91 kB 0 B
build/edit-site/style.css 3.91 kB 0 B
build/edit-widgets/index.js 26.3 kB 0 B
build/edit-widgets/style-rtl.css 3.13 kB 0 B
build/edit-widgets/style.css 3.13 kB 0 B
build/editor/editor-styles-rtl.css 476 B 0 B
build/editor/editor-styles.css 478 B 0 B
build/editor/style-rtl.css 3.85 kB 0 B
build/editor/style.css 3.85 kB 0 B
build/escape-html/index.js 735 B 0 B
build/format-library/style-rtl.css 547 B 0 B
build/format-library/style.css 548 B 0 B
build/is-shallow-equal/index.js 698 B 0 B
build/list-reusable-blocks/index.js 3.1 kB 0 B
build/list-reusable-blocks/style-rtl.css 476 B 0 B
build/list-reusable-blocks/style.css 476 B 0 B
build/nux/style-rtl.css 671 B 0 B
build/nux/style.css 668 B 0 B
build/rich-text/index.js 13.4 kB 0 B
build/server-side-render/index.js 2.77 kB 0 B
build/token-list/index.js 1.27 kB 0 B
build/warning/index.js 1.14 kB 0 B
build/wordcount/index.js 1.22 kB 0 B

compressed-size-action

@aduth
Copy link
Member Author

aduth commented Mar 6, 2020

It's expected this may also result in a smaller overall bundle size, as we are not using the full extent of the feature set of qs.

Also seen in the above #20693 (comment), there is a pretty nice drop. The results of npm run build before and after these changes are a difference of 11.1kb -> 5.5kb for the url bundle.

@aduth
Copy link
Member Author

aduth commented Mar 13, 2020

Relevant comment at #20738 (comment) about the commit 8dfe648 here.

As seen with the changes in 0a6046a, the new implementation is slightly more strict in regards to assuming the input argument to be a proper URL, not merely any string which happens to contain ?. While this could limit flexibility, I believe it's an appropriate change given:

Note that I had to scale this back a bit in #20693. Specifically, there's at least a few occurrences where we rely on an implementation like this:

addQueryArgs( '/wp-json/wp/v2/posts`, { type: 'post' } );

And if @wordpress/url only supports full URLs, it loses a lot of its usefulness.

The solution (8dfe648) is a bit hacky, effectively treating the input URL as the path of a throwaway base (second argument of URL constructor).

It's something I think we should want a consistent approach for in this module. Certain functions, we clearly want to expect a full URL (like isValidURL). Others, like getQueryString, addQueryArgs, removeQueryArgs, etc, we might want to be more tolerant about what's received, notably to support use-cases like in the quoted text above where we might want to be able to trigger fetch requests using an root-relative path.

Maybe the compromise is that for such functions, we specifically detect if the input is a root-relative URL and, if it is, construct URL with a fake base.

e.g.

let base;
if ( url.startsWith( '/' ) ) {
    base = 'https://example.com';
}

const urlInstance = new URL( url, base );

@sirreal
Copy link
Member

sirreal commented Mar 16, 2020

Maybe the compromise is that for such functions, we specifically detect if the input is a root-relative URL and, if it is, construct URL with a fake base.

e.g.

let base;
if ( url.startsWith( '/' ) ) {
    base = 'https://example.com';
}

const urlInstance = new URL( url, base );

There's some prior art for this approach by @sgomes. See https://github.com/Automattic/wp-calypso/blob/397d7592efea16154218b424602b15db64063361/client/lib/url/format.ts#L49 for an example.

@sgomes
Copy link
Contributor

sgomes commented Mar 16, 2020

There's some prior art for this approach by @sgomes. See https://github.com/Automattic/wp-calypso/blob/397d7592efea16154218b424602b15db64063361/client/lib/url/format.ts#L49 for an example.

Calypso's lib/url will likely become an independent package at some point. I expect it to have no dependencies but require a URL/URLSearchParams implementation present, either natively or through a polyfill.

I'm happy to accelerate any efforts in packaging it up if anyone's planning to reuse it!

@aduth
Copy link
Member Author

aduth commented Mar 16, 2020

@sirreal @sgomes That's interesting, thanks for sharing! I could definitely see some value in some of this, such as detecting the "type" of a URL. I think at least for the immediate purposes here, the current implementation proposed here more-or-less matches how it's implemented in Calypso's lib/url, though perhaps relying a little too much on the URL constructor to do the heavy lifting.

For me, it's largely a question of documentation. Some of the functions of this module demand to operate only with a full URL (isValidURL, getQueryString as of #20738 and then reverted here). In the case of isValidURL, it makes sense. So the uncertainty I have now is around (a) determining whether isValidURL is the sole exception, (b) documenting these expectations, and (c) implementing those expectations consistently.

@aduth aduth mentioned this pull request Mar 31, 2020
3 tasks
@aduth
Copy link
Member Author

aduth commented Apr 3, 2020

I just realized this pull request was left with the "In Progress" label. As far as I can recall, this is complete and ready for review.

@aduth aduth removed the [Status] In Progress Tracking issues with work in progress label Apr 3, 2020
Copy link
Member

@gziolo gziolo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rebased this PR to make it mergeable again. I plan to land it as soon as CI turns green.

@gziolo
Copy link
Member

gziolo commented Nov 30, 2020

@sirreal, any idea why TypeScript errors:

Error: packages/url/src/add-query-args.js(5,34): error TS6307: File ‘/home/runner/work/gutenberg/gutenberg/packages/url/src/build-query-string.js’ is not listed within the file list of project ‘/home/runner/work/gutenberg/gutenberg/packages/url/tsconfig.json’. Projects must list all files or use an ‘include’ pattern.

@sirreal
Copy link
Member

sirreal commented Nov 30, 2020

FYI @sgomes - removing qs dependency when working with URLs.

@gziolo
Copy link
Member

gziolo commented Nov 30, 2020

There is one failing e2e test but I saw it before in other PRs and it doesn't seem to be relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Package] Url /packages/url [Type] Performance Related to performance efforts
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants